…xtension

Implement the optional InstallExternalLibrary and UninstallExternalLibrary APIs
for the .NET C# language extension, enabling CREATE/DROP EXTERNAL LIBRARY support
for both ZIP-packaged and raw DLL C# libraries in SQL Server.

Implementation:
- Native C++ entry points in nativecsharpextension.cpp with null-runtime guards
- Managed C# implementation in CSharpExtension.cs using System.IO.Compression.ZipFile
- ZIP magic byte detection (PK header) to distinguish ZIP vs raw DLL content
- ZIP path: extract to temp dir, scan for inner .zip and extract to install dir;
  if no inner ZIP, copy files directly. Creates {libName}.dll alias when extracted
  file names differ from the SQL library name, so DllUtils can discover them.
- Raw DLL path: copy file to install dir as {libName}.dll
- UninstallExternalLibrary clears all files/dirs from the install directory
- Unique temp folder per call (Guid-based) to prevent race conditions
- Fix DllUtils.CreateDllList to search userLibName + ".*" pattern instead of
  exact name (pre-existing bug: exact name never matched files with extensions)

Testing:
- 14 new test cases in CSharpLibraryTests.cpp
- Updated ExecuteInvalidLibraryNameScriptTest for DllUtils pattern change
- 9 test packages in test/test_packages/
- All 73 tests pass (59 existing + 14 new)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

…fest-based cleanup, conflict detection, ALTER support

- Fix build script to use VS 2022 instead of VS 2019
- Add global.json to pin .NET SDK 8.0
- Fix: use library name as-is (no unconditional .dll append)
- InstallExternalLibrary: ZIP extraction with file conflict detection, manifest tracking
- UninstallExternalLibrary: manifest-based targeted file deletion, bottom-up empty dir cleanup
- ALTER EXTERNAL LIBRARY: clean up previous install before re-installing
- CHANGES.md documenting all changes

…LTER

New tests:
- ManifestWrittenTest, ManifestListsNestedFilesTest
- InstallLibNameAliasNoExtensionTest
- DirectoryOverlapAllowedTest, FileConflictFailsTest
- UninstallPreservesOtherLibrariesTest
- UninstallRemovesEmptyNestedDirsTest
- AlterExternalLibraryTest

Update 3 existing tests to reflect new libName-without-extension naming:
- InstallInvalidZipTest, InstallRawDllNotZipTest, InstallZipWithManyFilesTest

This PR branch is based on sicongliu's branch which predates PR microsoft#83 (DECIMAL Type Support on main). PR microsoft#83 added Microsoft.Data.SqlClient 5.2.2 as a required runtime dependency for SqlDecimal handling. Without it, sp_execute_external_script fails with HRESULT 0x80004004 when any script touches DECIMAL types.

When CREATE EXTERNAL LIBRARY is called with a name ending in .dll (e.g. [Scriptoria.dll] WITH (LANGUAGE='dotnet')), the raw-DLL install path and missing-alias fallback both unconditionally appended '.dll', producing files like 'Scriptoria.dll.dll' on disk. The CLR assembly resolver could not locate the assembly by simple name and ExtHost died with 'Could not load file or assembly Scriptoria'.

Added DllFileNameFor(libName) helper that returns libName as-is if it already ends in .dll (case-insensitive on Windows via existing s_pathComparison), otherwise appends .dll. Applied at the three sites in CSharpExtension.cs: raw-DLL install path, missing-alias fallback inside the ZIP install path, and UninstallExternalLibrary raw-DLL cleanup. ZIP-based installs whose library names do not end in .dll are unaffected.

Verified end-to-end on consumer side: Test-SpAiTaskSkills.ps1 reports 4 passed, 0 failed.

InstallRawDllWithDllSuffixedLibNameTest covers the case where the library is created via CREATE EXTERNAL LIBRARY [foo.dll] (libName already ending in .dll). Asserts that the raw DLL is written as 'foo.dll' (not 'foo.dll.dll') and that uninstall removes it from the same single-.dll path. Pairs with the DllFileNameFor helper added in this PR.

…e validation, recursive/conflict-checked alias, CopyDirectory no-overwrite

DllUtils.CreateDllList: try exact userLibName first; fall back to userLibName + '.*' only if no exact match. Extracted into AddMatches helper. Fixes callers that pass an explicit filename like 'Foo.dll' which previously became 'Foo.dll.*' and matched nothing.

CSharpExtension.UninstallExternalLibrary: call ValidateLibraryName before building manifestPath / libraryFile. Prevents malicious or legacy names with path separators from resolving outside installDir.

CSharpExtension.InstallExternalLibrary alias creation: (a) search the full extracted tree (not just top-level) for an existing '{libName}.*' before deciding to create an alias; (b) include the alias path in the conflict-check input so a collision fails BEFORE any content is written to installDir. Prevents partial-state failures when another library already owns '{libName}.dll' at the root.

CSharpExtension.CopyDirectory: use File.Copy overwrite:false (was overwrite:true) so TOCTOU changes between conflict-check and write fail loud rather than silently clobbering another library's files.

Tests: added UninstallRejectsPathTraversalLibNameTest and AliasConflictDetectedBeforeExtractionTest to cover the new behaviors.

…dirs ZIP guard, reparse-point sweep, libName extension-only rejection, refactor InstallExternalLibrary into helpers, doc + style sweep, 7 new tests

Block 1 substantive fixes:

- C1: AcquireInstallLock serializes Install + Uninstall on the same installDir via {installDir}/.install.lock (FileShare.None, DeleteOnClose, 100ms retry, blocks forever)

- C3: ValidateLibraryName rejects extension-only names (.dll, .txt, etc.) via Path.GetFileNameWithoutExtension

- E1: extractedFiles.Count==0 guard after collection prevents silent ALTER data loss when ZIP contains only directory entries

- E2: SAFETY comment on inner-zip ExtractToDirectory documenting why direct-to-installDir is safe today + tripwire for future .NET versions

- E3: New IsReparsePoint helper; root-level loops in Install + CollectRelativeFiles + CopyDirectory all guarded

Block 2 test fixes:

- E4: 5 new tests (AlterFromNonZipToNonZipTest, InstallRejectsInvalidLibNameTest, InstallZipWithDllSuffixedLibNameTest, UninstallWithMissingInstallDirTest, UninstallPreservesSharedNestedDirsTest) + new fixture testpackageJ-NESTED2.zip

- E5/E6/E7/E8/E10/E11: stronger assertions, removed redundant loops, exact equality, byte-for-byte content checks

- E12: surfaced + fixed latent test bug (dllCount expected 50 but real value with alias is 51)

- E13: InstallInvalidZipTest renamed to InstallNonZipFileAsRawDllTest

Block 3 refactor + style + docs:

- C6: InstallExternalLibrary split into orchestrator + 6 helpers (InstallRawDll, InstallZipPackage, FindInnerZip, CollectStagedFiles, DetermineAliasSource, ExtractContentToInstallDir, CreateAlias)

- A1-A8: empty lines before comments, single return, no var, brace style, scoped blocks unwrapped

- A5: s_pathComparer/s_pathComparison/s_lockRetryDelayMs moved to top class members

- B1-B5: full XML doc comments with <param> blocks; expanded IsZipFile + ValidateRelativePath comments

- C4: Install summary clarifies ZIP is allowed (not expected); raw DLL still supported

- C5: Logging.Trace -> Logging.Error in CleanupManifest skip path

- D1: <experimental/filesystem> -> <filesystem>

- D2: DirectoryHasFiles -> DoesDirectoryHaveFiles

- D4: Logging.Trace after manifest write (both raw-DLL and ZIP paths)

…111/111)

Found and fixed 4 issues introduced by the v3.9.0 changeset:

1. Lock file in installDir interfered with tests enumerating installDir contents and racing fs::remove_all on DeleteOnClose. Moved lock to a sibling path '{installDir}.install.lock' (concatenated, NOT a child of installDir).

2. C6 refactor lost a temp-folder cleanup invariant: 'tempFolder = InstallZipPackage(...)' only published the path on successful return, so a throw inside InstallZipPackage left an unreachable tempFolder leaking inside installDir. Converted to 'out string tempFolder' assigned BEFORE any work that can throw.

3. Three test bodies read files via std::ifstream without explicit close(); on Windows this raced CleanupInstallDir's fs::remove_all because the destructor runs lexically late. Added explicit .close() calls (RawDllInstallFailsIfForeignFileExists, InstallLibNameAlias, AlterFromNonZipToZip).

4. ErrorMessagePopulatedOnFailureTest mode 2 was checking for our ValidateRelativePath exception text, but .NET's ZipFile.ExtractToDirectory built-in zip-slip guard fires first with its own message ('outside the specified destination directory'). Updated assertion to match what actually surfaces.

Also: CMakeLists.txt was passing '--std=c++17' (a GCC/Clang flag, silently ignored by MSVC), so the test suite was actually building at MSVC's default C++14 the whole time. Switched to a generator expression that emits '/std:c++17' for MSVC and '--std=c++17' for non-MSVC compilers. Required to make D1 (std::filesystem) work.

Local test run: 111 passed, 0 failed (was 75 passed / 36 failed pre-fix).

…, A2-A7 correctness/robustness, B1 symlink regression test, doc + test cleanups

Bug / correctness fixes (Justin + yaelh review):

  A1 nativecsharpextension.cpp -- replace `new std::string(...)` +
     `c_str()` with `malloc(len+1)` + `memcpy` so the returned pointer
     owns its own buffer. Matches the managed `Marshal.AllocHGlobal`
     contract that ExtHost expects on the other side. Fixes UB where
     ExtHost would dereference a pointer into a freed `std::string`
     internal buffer.

  A2 CSharpExtension.cs -- narrow the `IOException` catch in
     `EnsureDir` via `when (IsSharingViolation(ex))` filter that
     inspects HResult for ERROR_SHARING_VIOLATION (32) /
     ERROR_LOCK_VIOLATION (33). Other IOException variants
     (DirectoryNotFoundException, PathTooLongException, mid-creation
     failures) and non-IOException exceptions (UnauthorizedAccess,
     Argument, Security) now propagate fast instead of being swallowed.

  A3 DllUtils.cs -- replace
     `Directory.GetFiles(searchPath, "{name}.*")` +
     `.Where(...EndsWith(".dll"))` with
     `Directory.EnumerateFiles(searchPath)` + explicit
     `Equals(".dll", OrdinalIgnoreCase)` on extension and
     `Equals(userLibName, OrdinalIgnoreCase)` on stem. No more reliance
     on Win32 `FindFirstFile` wildcard semantics; eliminates the
     "*.dll matches foo.dllx" and "Foo.* matches short-name 8.3 alias"
     over-match quirks. Remarks block documents both quirks for the
     next reader.

  A4 + B1 CSharpExtension.cs + CSharpLibraryTests.cpp +
     test_packages/testpackageK-SYMLINK.zip +
     test_packages/build-symlink-fixture.ps1 -- collapse the
     inner-zip and outer-zip install paths through a single
     `contentRoot` + `IsReparsePoint`-guarded
     `ExtractContentToInstallDir` helper. Both paths now extract the
     payload into `tempFolder/inner-content/` first and walk it on
     disk, instead of one path calling
     `ZipFile.ExtractToDirectory(innerZip, installDir)` directly.
     `CollectStagedFiles` simplified to a single on-disk walk;
     `ExtractContentToInstallDir` simplified to a single code path.

     B1 regression test `InnerZipFutureSymlinkRejectedTest` uses the
     new fixture (262 bytes, generated by `build-symlink-fixture.ps1`).
     Inner zip contains `legitfile.dll` (regular file) +
     `evil-symlink.dll` (Unix mode `0o120755`, content
     `/etc/passwd`). Today's .NET ZipFile ignores Unix mode bits and
     materializes the entry as a regular file, but a future runtime
     might honor them. Test asserts the future-proofing invariant:
     install succeeds, `legitfile.dll` lands in installDir, and
     installDir contains zero reparse points.

  A5 CSharpExtension.cs -- wrap `File.Delete(libraryFile)` in an
     `else` branch so the manifest path exclusively owns cleanup for
     current-version installs. Direct `File.Delete` only runs as a
     legacy-compat path for libraries installed by pre-PR builds that
     have no manifest. Comments explain the split.

  A6 CSharpExtension.cs -- add `s_reservedDeviceNames` HashSet
     (OrdinalIgnoreCase) at the top of the class with all 22 Windows
     reserved DOS device names (CON, NUL, AUX, PRN, COM1-COM9,
     LPT1-LPT9, plus the COM0/LPT0 historical aliases).
     `ValidateLibraryName` now rejects any libName whose stem
     (`Path.GetFileNameWithoutExtension`) matches. New rows in
     `InstallRejectsInvalidLibNameTest`: CON, nul, Aux, PRN, COM1,
     LPT9, CON.dll, nul.manifest. Rejection is enforced on every OS
     so behavior stays consistent for libraries moved between hosts.

  A7 CSharpExtension.cs -- `DetermineAliasSource` now picks the
     lexicographically-first `.dll` candidate
     (`string.CompareOrdinal`) instead of the first one returned by
     `Directory.GetFiles`. Stable across NTFS / ext4 / XFS and across
     re-installs.

  B6 CSharpExtension.cs + CSharpLibraryTests.cpp -- replace
     content-sniff `IsZipFile` with extension-based `HasZipExtension`.
     `InstallNonZipFileAsRawDllTest` (which asserted that a `.zip`
     file with non-PK bytes was silently rewritten as `{libName}.dll`)
     becomes `InstallZipExtensionWithBadContentFailsLoudlyTest` which
     asserts `SQL_ERROR` + that the user's file is NOT silently
     installed under a `.dll` rename. Pre-fix behavior would copy
     `bad-package-ZIP.zip` to `bad-package.dll`; post-fix the user
     gets a clear error.

  B12 CSharpExtension.cs -- first check in `ValidateLibraryName` is
     now `string.IsNullOrWhiteSpace(libName)`. New "   " row in
     `InstallRejectsInvalidLibNameTest`.

  B13 CSharpExtension.cs -- remove `ValidateRelativePath` entirely.
     After A4 collapsed both code paths through
     `ZipFile.ExtractToDirectory` + on-disk walk it had zero callers;
     the zip-slip defense it provided is now covered by (a)
     `ZipFile.ExtractToDirectory`'s built-in zip-slip check and (b)
     the on-disk walk being unable to see entries that escaped the
     staged tree.

Test additions / hardenings:

  B3 CSharpLibraryTests.cpp -- rename `DirectoryOverlapAllowedTest`
     -> `NonConflictingFlatFilesCoexistTest`. Comment block tightened
     to state "Both packages used here are flat (no nested
     directories), so the test exercises the flat-file coexistence
     case only -- nested-directory overlap is covered separately by
     `ManifestListsNestedFilesTest` + `InnerZipFileConflictFailsTest`."

  B4 CSharpLibraryTests.cpp -- harden `InstallZipWithManyFilesTest`
     with a per-module existence loop (`Module1.dll` ... `Module50.dll`)
     plus a comment block with the historical context (the original
     `EXPECT_EQ(dllCount, 50)` passed legitimately because the old
     install code created the alias as `{libName}` with no extension
     -- test asserted what the code did, not what it should do).

Documentation:

  B2 CSharpExtension.cs -- expand `IsReparsePoint` `<remarks>` block
     with the concrete `sneaky.dll` -> `/etc/shadow` scenario and the
     directory-level reparse-point variant. Notes the guard is
     theoretical against today's .NET (which writes symlink-mode
     entries as regular files) and links to
     `InnerZipFutureSymlinkRejectedTest`.

  B7 CSharpExtension.cs -- trim the `CleanupManifest` catch comment
     to stop at the diagnostic-trail justification.

Style:

  B5 / B9 / B10 -- blank line between adjacent independent constructs:
     (1) file-loop / dir-loop split in `ExtractContentToInstallDir`,
     (2) consecutive `if (e.find("MyLib.dll"))` /
         `if (e.find("native.dll"))` flag-setters in
         `ManifestListsNestedFilesTest`,
     (3) consecutive `if (e.find("testpackageA"))` /
         `if (e.find("testpackageB"))` flag-setters in
         `AlterFromZipToZipTest`.
     `}` followed by `continue;` guards or `return X;` function-tails
     left alone -- those are idiomatic and adding blank lines there
     would only add noise.

  B11 CSharpLibraryTests.cpp -- one-line rationale comments at the
     first write-site (`sentinelStream`) and the first read-site
     cluster (`aliasStream` / `sourceStream`) explaining the pattern
     of explicit `.close()` before `EXPECT_EQ` / `fs::exists` (gtest
     macro failure paths can run arbitrary code; known-closed streams
     are easier to debug). Same pattern is used at every other site
     for consistency.

Test results: 112/112 unit tests pass on Windows release config.

Production fixes (end-to-end testing against SQL Server 2025 RTM-GDR):

* CSharpExtension.cs AcquireInstallLock: place install.lock under
  Path.Combine(installDir, "install.lock") instead of one level up so
  concurrent installs into different <dbid>/<langid> slots don't
  serialize against one another.
* CSharpExtension.cs new DispatchAsZip(libName, libFilePath): install
  dispatch is now driven by the registered library name's extension
  (.zip -> ZIP, .dll -> raw DLL, otherwise fall back to libFilePath's
  extension). ExtHost passes a generated temp filename with no semantic
  suffix in production, so the previous extension-sniff on libFilePath
  was meaningless there; the libFilePath fallback preserves the legacy
  contract for test fixtures registered under a bare library name.
* CSharpOutputDataSet.cs + utils/Sql.cs DotNetNVarChar plumbing: add
  the DotNetNVarChar row to Sql.DataTypeSize and a DotNetNVarChar case
  alongside DotNetWChar in ExtractColumn / GetStrLenNullMap. Previously
  fell through to default and threw KeyNotFoundException in
  DataTypeSize before the column reached the dispatch switch.
* CSharpOutputDataSet.cs DotNetWChar / DotNetNVarChar Size unit:
  report Size in BYTES, matching the unit emitted by GetStrLenNullMap
  (Encoding.Unicode.GetByteCount). The previous code reported a
  character count, which combined with a byte-count length map caused
  SPEES to log "Reading one row failed for column N row M. The length
  information is incorrect." and reject the rowset whenever a string
  column contained non-ASCII data.

Reviewer items (PR microsoft#85 May 13 review):

* test/src/native/CMakeLists.txt: non-MSVC -std=c++17 (one dash) instead
  of --std=c++17 to match the documented spelling and the rest of the
  build tree.
* include/nativecsharpextension.h: rewrite InstallExternalLibrary doc
  comment to spell out the new libName-based dispatch contract and the
  {libName}.manifest file written for every install (ZIP or raw DLL).
* CSharpExtension.cs DetermineAliasSource: alias-suppression now
  requires an EXACT match against "{libName}.dll" at the install root.
  The previous prefix check ("{libName}.") suppressed alias creation
  for ZIPs that planted only sidecars at the root (foo.deps.json etc.)
  with the real binary nested under lib/net8.0/, leaving the install
  un-loadable. Drops the now-unused libName parameter from the
  signature. Pinned by AliasCreatedWhenOnlySidecarsAtRootTest +
  testpackageL-SIDECAR.zip fixture (build-sidecar-fixture.ps1).
* test/src/native/CSharpLibraryTests.cpp new FreeLibError(SQLCHAR *)
  helper using LocalFree (matches the production Marshal.AllocHGlobal /
  LocalAlloc allocator that ExtHost uses on the consumer side). Wired
  into CallInstall, CallUninstall, and CallInstallCaptureError so the
  test harness no longer leaks the libError buffer on every
  failing-install assertion.

Tests: 113/113 unit tests pass on Windows release config (one new
TEST_F: AliasCreatedWhenOnlySidecarsAtRootTest).